If you're training a robot, a world model or an AI that runs on glasses or a headset, you need video shot from where the person doing the task is standing, with their hands in frame. That's egocentric video, also called POV or first-person video, and it's scarce because almost everything ever filmed was shot from a tripod, a drone or a phone pointed at someone else. Read on to find out what an egocentric dataset is, who trains on it, what a good one has, the rights that come with it, and where to get it.
What egocentric video data is
Egocentric video is footage recorded from a camera mounted on the person, usually on the head or the chest, so the frame shows what they see as they act. The hands are in view, the task unfolds in order, and the camera moves the way a body moves. Third-person footage (exocentric, in the research vocabulary) shows the same task from outside, and it teaches a model what an action looks like without teaching what it feels like to do it.
That difference is why egocentric data exists as its own category. An embodied model needs to learn what the world does when you act on it (how an object responds to force, how a task breaks into steps, how a hand and a tool work together). Standard internet video captures very little of that. Our guide to training data for robotics and physical AI covers the other kinds of data in the stack.
Who trains on egocentric video
Four kinds of team buy it, and they want slightly different things.
Robotics and embodied AI teams want the closest thing to letting a robot see through human eyes: hands, objects, tools and the steps of a task, from the viewpoint a humanoid or a manipulator deploys in. They use it to pretrain before teleoperation data takes over for the last mile.
World model and video generation teams want continuous first-person footage of the physical world, because a model that predicts what happens next needs to have watched a lot of things happen from inside the scene. First-person video generation is a use case in its own right. Our guide to gameplay data for world models covers the simulated side of the same need.
AR and wearable teams want footage from the device's own viewpoint (glasses, headsets, body-worn cameras). Egocentric action recognition, action anticipation (what will this person do next) and hand-object interaction modeling all start here.
Computer use and device teams want a narrower slice, people picking up phones, tablets and laptops and switching between a screen and the physical world, so a model learns the context around the interface, not just the interface.
What a good egocentric dataset has
| What to look for | Why it matters | What to check in a sample |
|---|---|---|
| Camera on the body, at natural eye or chest height | The viewpoint has to match the one the model will deploy in | Mount type is recorded per clip, and the framing matches it |
| Hands and objects in frame | Hand-object interaction is the signal most teams are paying for | A flag per clip for whether hands and objects are visible |
| Tasks completed start to finish | A model learns the structure of a task from whole episodes, not fragments | Clip length, and whether clips are cut mid-task |
| Continuous, naturalistic activity | Staged, pre-segmented demonstrations teach a narrower distribution than real life | Whether the footage was scripted or captured during ordinary activity |
| Variety of environments and activities | A kitchen-only set trains a kitchen-only model | Counts of environments and activity types, and the split between them |
| Metadata you can filter on | You'll want to pull every clip of one activity in one setting without watching all of it | Fields for duration, mount, environment, activity type and hand visibility |
| Documented consent per recording | The person wearing the camera, and the people in frame, have to have agreed to AI training | Per-asset rights documentation, not a blanket statement |
Hours matter less than the rows above. A few thousand hours with hands in frame, whole tasks and clean metadata beat a much larger pile of head-mounted footage of someone walking to the bus.
Where to get egocentric video data
Three routes, and most programs use more than one.
Public research sets, like Ego4D and EPIC-KITCHENS, are where the field started. They're free, well documented and everyone has trained on them, and that is the limit. They add little that competitors don't have, and each license has to be read before commercial training. They're the right place to prototype.
Collecting your own means putting cameras on people and running tasks to a brief. You get the viewpoint and the tasks you want, and you pay for every hour, the recruiting, the consent paperwork and the annotation. It's the route for a task nothing on the market covers.
Licensed catalogs are footage that already exists, recorded by people going about ordinary and specialized activities, cleared for AI training by the people who filmed it, and delivered with metadata. This is the route for scale and variety at a known price. Our guide to licensed video data for AI training covers how licensed video is priced and delivered, and our guide to robotics data collection companies and providers covers who supplies each route.
The rights in first-person footage
Egocentric video stacks more rights than a tripod shot. The person wearing the camera owns the recording and has to have consented to AI training by name. Everyone who walks into frame has their own likeness and, in some places, biometric rights. Locations can carry restrictions of their own (a workplace, a store, a school). And a body-worn camera records audio by default, which brings speech and voice rights with it.
For a buyer, the test is the same as for any training data. For each recording, can the provider show a document in which someone with the authority to do so licensed it for AI training? Our guide to rights-cleared training data covers what that documentation looks like and what separates real clearance from a label.
What's in Troveo's POV catalog
Troveo licenses egocentric footage recorded by more than 7,000 rights holders, cleared for AI training with documentation per asset, and ships it cleaned and normalized with metadata. Two of the datasets are first-person by design:
- Everyday Tasks POV: more than 12,400 hours of head- and chest-mounted footage of routine activity (cooking, cleaning, tool use, work, personal care, navigation) across more than 20 activity categories and 10 environments, including home, office, retail, transit and outdoor settings. Naturalistic continuous activity, not staged demonstrations. Metadata per clip covers duration, camera mount, environment, activity type and whether hands and objects are in frame.
- Device Usage POV: more than 2,200 hours of first-person footage of people picking up, navigating and operating phones, tablets, laptops, smart-home devices and wearables, across more than 12 device categories and 15 settings, including the switching between physical objects and on-screen tasks.
Around them sit the related physical AI sets: human-object interaction, tool manipulation, hands-only manipulation and task demonstration, which cover the same activities from close-up and third-person viewpoints for teams that want both.
How to evaluate a sample before you buy
Ask for a sample that matches your deployment, then check five things:
- Watch ten clips end to end and count how many show a task from start to finish.
- Filter the metadata for one activity in one environment and confirm the clips that come back match the labels.
- Check the hands-visible flag against what you see.
- Ask for the rights documentation on three specific clips, not the catalog as a whole.
- Run the sample through your own preprocessing and see what breaks before you commit to volume.
Our guide to how and where to buy AI training data covers the rest of the buying process.
Where Troveo fits
Troveo is a licensed data marketplace, and first-person footage of people doing physical things is one of the deepest parts of the catalog. More than 7,000 rights holders, more than 20 million dollars paid out, per-asset documentation, delivered training-ready. Browse the POV and physical AI categories in Troveo Lens, or send us the task list your model needs to learn and we'll put samples in front of you.