Vision–Language–Action Research Log

The Introduction

The first time VLAs caught our attention was the 'Robotics: Why Now?' talk on the AI Engineer's YouTube channel. It stood out because it felt like the first time someone clearly explained how to solve the generalization problem in robotics without needing years of hard-coded rules.

The idea that we are finally at a point where we can train a physical policy the same way we train a language model was something we needed to validate for ourselves. We got pretty obsessed with the idea of taking these research concepts and seeing if we could actually make them work on the limited hardware we have available.

We did not want to just read the papers and assume the math works. We wanted to go through the entire pipeline from calibration to data collection and see where the actual friction points are when you do not have a perfect lab setup.

We decided to test out the current state-of-the-art open source tools to see what is actually usable right now. This document is a log of that process and details how we navigated through broken implementations to eventually hack together a working pipeline that trained a real policy.

Simulation and Model Exploration for Laptop-Scale VLA Systems

1. Constraints

A core objective of our project was to explore the feasibility of running Vision–Language–Action (VLA) style pipelines on consumer-grade hardware, specifically a personal laptop, alongside experiments on a 3D printed robotic arm.

A key constraint was hardware availability. We did not have access to a high VRAM GPU (≥16–24 GB), which ruled out heavyweight simulators and large end-to-end VLAs commonly used in recent literature. This constraint strongly influenced our choice of simulators, learning paradigms, and model scale.

2. Exploration of Simulation Environments

We evaluated multiple robotic simulation environments with the goal of identifying platforms that are lightweight, suitable for manipulation tasks, and compatible with learning-based control pipelines.

2.1 Excluded: Isaac Sim

Although Isaac Sim is widely used for robotic manipulation and sim-to-real workflows, it was excluded early due to very high GPU VRAM requirements, heavy reliance on RTX rendering, and unsuitability for laptop-scale experimentation.

2.2 CoppeliaSim (Imitation-Learning Focus)

CoppeliaSim was explored due to its relatively low computational footprint, mature support for robotic arms and sensors, and ease of integration with Python and ROS.

However, our experiments highlighted that CoppeliaSim is better suited for imitation learning or scripted control pipelines rather than full end-to-end VLA training. Language grounding is not natively supported, and vision–language alignment must be handled externally.

2.3 ManiSkill

ManiSkill was investigated for its focus on manipulation benchmarks and structured task definitions. While well-suited for learning-based manipulation, it remained computationally heavy for full VLA pipelines on laptops. Language integration exists but is indirect and task-specific.

2.4 CRISP

CRISP was explored as a language-conditioned control framework emphasizing lightweight architectures, modular design, and compatibility with imitation learning. It aligned well with our hardware constraints and reinforced the practicality of modular VLA approaches.

3. Shift in Learning Paradigm: Full VLA to Partial VLA

From simulator experimentation, a key insight emerged: end-to-end VLA models are currently impractical on consumer laptops. Modular or imitation-based approaches, like the leader–follower setup of LeRobot, are far more feasible and scalable under limited compute.

4. Literature Review and Model Scaling

Parallel to simulator exploration, we conducted a focused literature review on small and efficient VLA models, including smolVLA, tinyVLA, and pi-zero style models. These works emphasize policy distillation, offline imitation, and language-as-context strategies over monolithic end-to-end grounding.

Building the SO-101 Arms

Looking for Existing Arms

After exploring simulations, we began by exploring whether we could get access to existing robotic arms in IIT Bombay that could be used to collect demonstration data and fine-tune a compact VLA model.

After a lot of tries, waiting, and exploring labs and student teams at IITB, we got access to a 6-DoF arm built by the Gensus team, IITB. However, we realised that it had continuous rotation servos, meaning position control without attaching encoders on each joint was going to be very tough. We then also got access to a Dobot arm in the ARMS lab at IITB.

However, this Dobot arm was not completely assembled and the people at ARMS lab were very slow in collaborating. So, due to limited hardware availability and missing components, this path proved unsustainable.

Deciding to Build it Ourselves

Rather than slowing down, we made the decision to build our own robotic arm from scratch.

We adopted the LeRobot SO101 open-source design and immediately procured the STS3215 serial motors on the same day, travelling nearly three hours by local train to avoid delays.

With no prior hands-on experience, we rapidly learned practical 3D printing, from slicer configuration to filament behavior. After a failed first print, we iterated and mastered critical parameters such as wall thickness, tolerances, infill density and patterns, and part orientation for optimal strength, time, and surface quality.

By fully understanding and optimizing Creality Print workflows, we successfully fabricated and assembled the complete arm within just five days.

Calibration and Teleoperation

We first calibrated the SO-101 using the official LeRobot calibration script and tried to run the example teleop to validate hardware. The built-in teleop expects either a leader arm or a gamepad leader, which revealed why our follower-only setup was not behaving as we wanted.

When the default teleop did not match our workflow, we dug into the code and wrote a custom keyboard teleop controller to directly command all six DOFs. That let us trial joint-space control and rapidly iterate on smoothing and rate limits. It worked well enough to show simple demonstrations and taught us the practical limits of simple joint-by-joint keyboard control versus coordinated trajectories.

We then discovered a helpful fork called XLeRobot which packages SO-101 examples, teleop utilities, and even a YOLO-based object-following demo. Reading their code was instructive. The scripts are straightforward and directly applicable to the SO-101, so we could adapt ideas, especially the keyboard-joint control and simple object-follow loop, instead of writing the code ourselves. That saved time and validated our approach.

Because data format and ease of dataset collection matter a lot, we then found Phospho, their web UI and starter pack.

Phospho provides a polished web interface and multiple teleoperation modalities (gamepad, keyboard, VR/Quest, leader/follower APIs) that make recording episodes and inspecting logs much faster than a raw script-only workflow. The UI made multi-method teleoperation and quick dataset inspection much easier.

After reading discussions on LeRobot's Discord server, we realised that we needed at least two cameras and one of them should preferably be close to the gripper.

We did not have a second dedicated camera, so we improvised: a USB webcam on the wrist and a phone as a second camera tunnelled to the laptop as a virtual camera. We used simple phone-as-webcam techniques with OBS and VDO.Ninja workflows to make the phone feed appear as a regular camera input.

This was also crucial because the literature and practical projects consistently show that a wrist camera plus an external viewpoint is extremely helpful for closed-loop manipulation learning, because the wrist view captures local contact details while the external view gives overall context. Adding a wrist camera markedly improved the quality of the visual input for training.

To inspect what we had collected, we used and learned from the LeRobot dataset visualizer tools on Hugging Face and GitHub Spaces. The visualizer is an interactive way to play back synchronized videos and telemetry so you can quickly see issues like noisy joint targets, missed frames, or misaligned timestamps.

The dataset we collected was noisy, as expected from human teleoperation without leader guidance, but improved quickly after a few episodes. Teleoperators naturally learn smoother, more coordinated motions and the resulting trajectories look much more human, not one DOF at a time but coordinated multi-DOF motion.

That human learning curve is something we watched and quantified with the visualizer. Episode by episode the jerkiness decreased and the replayability rose, which gave us confidence to move from data collection to model training.

We then studied the Action Chunking with Transformers (ACT) paper, introduced with the ALOHA system, which proposes predicting short sequences of future actions, or action chunks, instead of single-step controls. The model is implemented as a Transformer-based conditional VAE with temporal ensembling, a design choice that significantly reduces compounding errors in long-horizon manipulation and enables smoother, more stable behaviors from limited teleoperated data.

The paper also empirically demonstrates the importance of multi-view visual input, particularly the use of wrist-mounted cameras, for real-world fine manipulation.

Guided by these insights, we trained an ACT-style policy on our collected demonstrations and further fine-tuned it using reinforcement learning.

Computer Vision Teleoperation

In parallel with the hardware build, we focused on enabling rapid and scalable data collection through a custom computer-vision-based teleoperation pipeline.

By closely following technical discussions on X and analyzing numerous robotics and VLA-focused YouTube demonstrations, we identified a clear pattern. CV-based teleoperation is a critical enabler for efficient, low-cost demonstration data collection. These insights strongly influenced our system design choices.

Instead of relying on expensive motion-capture systems or proprietary controllers, we implemented a lightweight setup using a standard webcam and real-time hand tracking for intuitive human-in-the-loop control. We developed normalized motion mapping and velocity-based control to ensure stable arm behavior.

To mitigate noise and jitter inherent in vision systems, we introduced temporal smoothing and filtering, resulting in consistent and repeatable trajectories. This CV TeleOp pipeline significantly reduced setup complexity, operator fatigue, and data collection time, while producing high-quality demonstrations suitable for VLA training, reflecting our emphasis on learning from the broader research community and rapidly translating insights into working systems.

To further improve demonstration quality, we implemented a dedicated motion-smoothing layer between the CV TeleOp inputs and the robot's low-level controllers.

Raw vision-based hand signals are inherently noisy and can introduce high-frequency jitter, which negatively impacts both hardware safety and data usability for VLA training.

To address this, we applied temporal filtering and velocity smoothing to convert discrete, frame-level hand movements into continuous, physically plausible trajectories.

This smoothing module enforces gradual acceleration and deceleration, limits sudden spikes in joint velocities, and preserves intent while removing noise.

As a result, the robot executes fluid, human-like motions that are safer for prolonged operation and far more consistent across repeated demonstrations.

Beyond improving teleoperation experience, this step significantly increases dataset quality by producing clean, learnable trajectories, reducing post-processing effort, and directly enhancing downstream VLA training stability and performance.

The Trip to ROSCon India

As a team, we also made an effort to learn from the broader robotics community. One of our members based in Pune attended ROSCon India, which turned out to be unexpectedly aligned with our interests. A significant number of talks directly or indirectly touched on VLAs and Physical AI.

The opening keynote by Mihir Acharya from NVIDIA set the tone. He framed Physical AI as the natural progression from perception AI to generative AI to agentic AI, motivated by real-world pressures like manufacturing reshoring and labor shortages.

He gave a clear overview of NVIDIA's ecosystem: Isaac Sim for simulation, Omniverse and Cosmos for high-fidelity sim-to-real data generation, and GR00T for deployment and training of generalist robot policies. He also briefly mentioned pretrained and synthetic pipelines such as Cu-Motion and Synthetic DETR, which highlighted how much of the data and training stack is becoming standardized and accessible.

A talk by Lentin Joseph on the future of ROS 2 in the age of Physical AI echoed this shift, suggesting that the ROS ecosystem is gradually moving from Gazebo-centric workflows toward Isaac-based simulation as VLA-style learning and large-scale data generation become more common. This reinforced our sense that learning Isaac-centric tooling is becoming increasingly important for modern robotics research.

One of the most relevant talks for us was by Yadnesh Chakne from Apart Research, who gave a clear overview of how the VLA space has evolved, from the Open X-Embodiment dataset and large-scale multi-robot data collection, to ideas like the Moravec paradox and the role of action chunking in transformers for stabilizing long-horizon control.

This directly connected with our own experiments and helped contextualize why chunked action prediction is becoming a standard design choice.

On the second day, an industry talk from Universal Robots showcased their next-generation AI Accelerator Package, emphasizing how industrial arms are moving toward semi-autonomous and learning-based operation.

A particularly insightful session from Clutterbot discussed their attempt to build a small general-purpose household robot, somewhere between a Roomba and a humanoid.

Their experience strongly reinforced two practical lessons we have also seen firsthand. High-quality teleoperation matters far more than simple keyboard control, and dataset cleanliness is critical for training usable VLA policies.

Overall, ROSCon India made it clear to us that VLA and Physical AI are not niche ideas but active, fast-growing areas with both academic and industrial traction.

The conversations and talks helped validate our direction, exposed us to tools and workflows we now plan to explore further, especially Isaac Sim, Omniverse, and Cosmos, and reinforced the importance of good data, good teleop, and scalable learning pipelines.

The Plan Ahead

Tactile Encoder

In the next phase, we plan to extend the Vision–Language–Action architecture by integrating a tactile encoder to enable contact-aware and force-sensitive manipulation.

The core idea is to treat tactile sensing as a first-class modality, alongside vision and language, rather than as a post-hoc feedback signal.

We will instrument the end-effector with high-resolution tactile sensors, such as pressure or capacitive arrays, and stream time-synchronized tactile observations during interaction-heavy tasks such as grasping, insertion, and surface following.

From an architectural standpoint, raw tactile signals will be encoded using a lightweight temporal encoder, such as a 1D convolutional stack or a small Transformer, to capture contact dynamics, force gradients, and slip events over time.

The resulting tactile embeddings will be projected into the same latent space as visual tokens and fused using cross-attention, allowing the policy to condition actions jointly on what it sees and what it physically feels.

This multimodal fusion will occur before action decoding, enabling the model to resolve visual ambiguity through touch, for example detecting grasp success or object compliance.

During training, tactile-conditioned trajectories will improve credit assignment for contact-rich behaviors, reducing reliance on vision alone and increasing robustness to occlusion and lighting changes.

At inference time, the tactile encoder will operate in closed loop, allowing the VLA policy to adapt actions online based on real-time contact feedback.

Cheaper Leader for Better Data Collection

To significantly reduce system cost while preserving high-quality demonstration data, we propose a novel leader–follower setup in which the SO-101 leader arm replaces active servos with precision potentiometers.

Instead of actuating the leader arm, each joint is instrumented with a rotary potentiometer that directly measures joint angles as the human operator moves the arm manually.

This eliminates the need for torque-capable motors, motor drivers, and power electronics on the leader side, resulting in a substantial reduction in hardware cost and complexity.

From a systems perspective, potentiometer readings are sampled at high frequency using a microcontroller ADC, calibrated to map voltage ranges to joint angles with low quantization error.

These joint states are streamed in real time and used as supervisory signals for the follower arm, which executes the motions using actuated servos.

This architecture cleanly decouples sensing from actuation, minimizes latency, and avoids backdrivability constraints imposed by motorized joints.

Beyond cost savings, this approach improves safety, simplifies maintenance, and enables scalable data collection setups, making high-quality imitation and VLA datasets accessible without expensive teleoperation hardware.