How grasys does AI-driven development: LLM routing, TDD test guardrails, and Human-in-the-loop

Since its founding in 2014, grasys Inc. has supported the high-traffic business infrastructure of enterprise companies serving hundreds of millions of users—from building and operating cloud infrastructure to developing and operating applications on the cloud—through robust engineering.

The company's technical strength is demonstrated in both infrastructure construction using IaC (Infrastructure as Code) and agile application development. In particular, amid the dramatic shift in development paradigms brought about by the rise of generative AI in recent years, the company was quick to place AI at the core of its workflows.

Following the previous article on infrastructure engineering using "Policy as Code," this installment focuses on "grasys-style AI-Driven Development." We conducted an in-depth interview with Tetsuya Shinchi, Chief of the Cloud Development Sec., who leads development at the front lines, covering everything from LLM (large language model) routing strategies, to secure context sharing with AI, to the implementation of test harnesses that prevent AI from running out of control through test-driven practices, and the organization's approach to "Human-in-the-loop."

Tetsuya Shinchi, Chief of Cloud Development Sec., grasys Inc.

――Today, we would like to explore grasys's AI-driven development efforts through this interview. Without further ado, could you first tell us about the evolution of your own career and when you joined grasys?

Shinchi: I have built my career as a full-stack engineer at multiple workplaces, and I joined grasys in April 2024. Immediately after joining, I have been involved in internal platform development projects and in the operation and development of large-scale systems entrusted to us by client companies. When I joined, the use of AI in development workplaces was on the verge of becoming commonplace across society as a whole—it was precisely a time when the very concept of development was beginning to change dramatically.

――As technology changes—specifically, as the cloud evolves and AI becomes more widespread—the boundary between infrastructure and development is becoming blurred. Could you tell us about the background and timing of grasys's decision to fully incorporate AI into its development processes?

Shinchi: Around the time I joined in 2024, I believe there was a company-wide push to introduce AI into development processes all at once. From my own personal experience as well, starting around January or February 2026, conversations began to emerge among engineers along the lines of "if you practice harness engineering, you barely have to write anything by hand." When I actually tried incorporating AI agents into my own development environment, manual coding and rework decreased dramatically, so from around February of that year I shifted fully to a development style centered on AI.

Previously, everything from the basic architecture design to the coding of logic was typed and built from scratch by human engineers, but now I feel that AI has taken over the majority of the physical work of "writing code."

――You joined grasys and are now at a workplace that actively adopts AI. Could you tell us about the "overwhelming benefits" you actually experience at the development front lines, and conversely, any "technical barriers or organizational challenges" you have faced?

Shinchi: First, as clear benefits, advanced autocompletion in the IDE (integrated development environment) has become possible, and simply giving instructions in natural language through a chat interface now enables complex environment operations and refactoring.

For example, when using a completely new feature or API provided by a cloud provider, we can load the entire official documentation (PDFs or references) into the AI's context and ask, "How do we implement this in our environment?" and it will propose appropriate code implementations all at once and provide implementation samples. This is revolutionary from the perspective of cognitive load as well.

On the other hand, what I feel as a disadvantage (challenge) is the concern that "the fundamental skill improvement of engineers," including younger engineers, may be dulled. Previously, by glaring at official documentation while errors were thrown and struggling through debugging, we built a deep "mental model" of the system.

However, now that process is largely skipped, and people no longer have to struggle more than necessary, making it easy to fall into a state where "for now, if you leave it to AI, you can get something working even while the internal structure remains a black box." As a result, I think the part where the opportunity to "learn the essentials of architecture and the foundations of technology through painstaking effort" has decreased may become a future development challenge.

LLM routing strategy: Policy for selecting AI models according to the task and token optimization

--In development, among the many LLMs (large language models), by what criteria do you select and route models?

Shinchi: Rather than "always using the latest heavy model," we clearly use different AI models depending on the nature and complexity of the task. For example, for tasks where rules and syntax are to some extent textualized and the work content is limited and clear, such as simple coding or refactoring, we use relatively older-generation lightweight models. This is because even if a large amount of code is read and token consumption increases, the cost does not skyrocket.

On the other hand, when defining requirements from scratch, when we are stuck and cannot arrive at the optimal architecture ourselves, or when we want ideas that exceed assumptions from a completely different perspective, we intentionally choose the latest and most intelligent model with the highest reasoning capability.

Currently, we use OpenAI's ChatGPT and Google Cloud's Gemini, etc., in parallel, but the latest Gemini models in particular have the strength of being able to process ultra-long-context inputs, so we use them to comprehensively reference and analyze not only each client's infrastructure configuration and the history of how the current architecture came to be, but also internal documents on similar systems for other clients, and to formulate future design policies based on multifaceted data.

Another point is that grasys is also a company strong in development based on Google Cloud, so in that respect I think there is good compatibility with Gemini.

At grasys, in taking charge of clients' critical systems, please tell us about security when passing context (domain knowledge) to AI and about managing access permissions (IAM).

Shinchi: As a major premise, as a company policy, permissions regarding "who can access which resources and data" are strictly centrally managed through IAM (Identity and Access Management) and similar mechanisms. The environment in which AI agents run is no exception, and it fully complies with this permission design. In other words, the architecture ensures that information such as company confidential information or specific customer data that the individual engineer does not originally have access rights to can never be passed to the AI side (and cannot be accessed). Thanks to this guardrail, we can minimize the risk of data leakage and fully utilize AI with peace of mind in a secure environment.

Shinchi: Actually, that is the current major bottleneck in AI-driven development. Because AI outputs code at ultra-high speed, cases are becoming apparent as an issue in which engineers themselves, without deeply understanding the logic, treat it as "for now it works locally, so it's fine" and send it to a Pull Request (review).

At first glance, even if the implementation phase appears to have finished at explosive speed, I think several problems can occur when it is checked at the reviewer stage.

For example, points such as "in refactoring work, making superficial fixes, or functions deleted in refactoring being left as they are elsewhere," "as a result of insufficient understanding of the purpose, making one function perform multiple tasks," and "lack of comments for humans." The cause of these things happening is that the side using AI does not understand in the first place what they want the AI to do.

Normally, from a Human-in-the-loop perspective, before going through review (within the scope of an individual engineer's responsibility), one must fully understand the AI's output and take code ownership before running it. This point—that "the speed of human confirmation and understanding cannot keep up with the speed of AI generation"—has become a major practical barrier.

――To prevent this "human oversight" and "AI quality fluctuation," what kind of workflow (development process) does grasys actually incorporate AI into and control in real development settings?

Chinji: As the greatest mechanism for preventing AI from running out of control, we have adopted a flow that combines "Issue Driven" with "Test-Driven Development by AI (TDD)."

For example, on GitHub, we first create an Issue such as "Implement a product search function on a demo e-commerce site" in Japanese at a resolution that is easy for AI to understand. This is where it differs from the conventional method of having AI write implementation code right away: before implementation, we give the AI an instruction such as "Create a feature branch from develop, and implement the feature using TDD according to tdd-playbook. The target Issue this time is #100. Please start from Step 1."

Next, once the test cases have been created, we set it so that the system enters a state of waiting for human approval of the test case contents. Here, after human approval, the flow moves to the implementation steps described in the skill, and the AI begins implementation.

The AI then reads and interprets the contents of the Issue and autonomously creates, in Japanese, the framework of test cases necessary for implementation. Engineers strictly review, first of all, not the generated implementation code itself, but the specifications and logic of the test cases defined in Japanese.

If the test cases match completely with the business requirements we had assumed, we commit the framework of those test cases to the repository first.

Chinji: Once the test cases satisfying the specifications are finalized, we then tell the AI "approve the test cases." The AI starts the subsequent implementation steps based on the finalized test cases as an "absolute framework of correct answers," so engineers simply wait for the AI to complete implementation while doing other work. In terms of actual work, this cycle is repeated at high speed.

Before entering the implementation phase, by having humans insert one review at the test case stage (moving Human-in-the-loop earlier = shifting left), we lay down strong guardrails so that the AI's actions do not proceed in a wrong direction deviating from the requirements.

Instruction to create a GitHub Issue from the IDE (click image to enlarge)

Issue created by AI (click image to enlarge)

Generation and approval of test cases (click image to enlarge)

Test code is implemented according to the test cases (click image to enlarge)

Comparison of generation results before implementation (left) and after implementation (right) (click image to enlarge)

However, with this method, because the requirements are thoroughly shared between humans and AI at the test case stage, even if the AI outputs a temporary error during the implementation phase, the AI itself can notice that error through the tests and autonomously make repeated corrections. Therefore, engineers do not need to reissue detailed instructions each time, and can leave it to the AI to autonomously complete implementation until all tests pass and the requirements are satisfied.

As a result, because an implementation satisfying the requirements is guaranteed by the time humans perform review, rework is eliminated, and the personalization/dependence on specific individuals in review can be resolved at the same time.

grasys-style "knowledge sharing" and the evolution of the engineering organization

Chinji: The way we learn technology and the process of absorbing knowledge (Developer Experience) have changed fundamentally with the introduction of AI. Previously, when learning a new middleware or architecture, we read thick official documentation or PDF technical books from top to bottom. But now, we have come to feel that it is extremely tiring (inefficient) for humans to visually follow and read such static PDFs with fixed formats.

As an alternative approach, we began using a RAG (Retrieval-Augmented Generation)-style method: loading the entire target document into the AI chat engine as context and prompting it with "Within the requirements of this project, summarize only the configuration items that require attention and the necessary parts." This put us in a state where the AI dynamically extracts and tells us just the "points that should be read" in line with the context of our own environment, and the speed of technical verification improved dramatically.

Chinji: So that we don't depend on a particular individual as the "AI operator," within the team we have set up a dedicated repository, in the manner of an "Internal Developer Portal," for sharing convenient AI operation scripts that someone created and that worked well, as well as system prompt templates for injecting specific context. We leave markdown documents there in Readme format, establishing a "highly reproducible mechanism" that lets other members immediately use AI in the same environment with the same accuracy.

What grasys values is defining the guardrails necessary for the organization while giving the front line greater discretion. Rules that must be followed and important policies are clearly indicated by management and managers, and within that scope we create an environment in which each engineer can think, judge, and devise for themselves. Rather than deciding the way of proceeding uniformly down to the details, we emphasize making the most of the ideas and possibilities that arise from the front line.

In AI utilization as well, we believe it is important not to make delegating itself the goal, but to discern, according to the purpose of the work, which parts to leave to AI and which parts people should judge and confirm. We explore ways of use that lead to better outcomes, including quality and safety as well as the burden of confirmation and operation. For that purpose, discretion that allows the front line's knowledge and experience to be put to use is necessary.

What we aim for is to solve customers' challenges with technical capability and realize that purpose. AI is one of the means we use for that. Even when using AI to implement code or summarize the key points of documents, we value engineers themselves understanding and judging the content and intent. We think about what is necessary for the customer from both the cloud infrastructure and application sides, and connect that to design and implementation. Effectively utilizing AI in that process is what grasys considers AI-driven development.

At grasys, we want to work together with engineers who face customers' challenges and purposes, think for themselves, choose technologies, and work toward realization. We want to use AI as a means for that as well. If, after reading this article, you feel "this sounds like a good fit for me," please do get in touch.

――Thank you very much.