Welcome to the world of automation powered by AI


With one API call, you can add the power of AI to your mobile test automation. The team at test.ai has teamed up with Jonathan Lipps, the lead contributor of Appium and founder of Cloud Grey, to add a bit of AI to Appium. The AI finds common elements in mobile apps such as search text boxes, login buttons, etc., so test developers don’t have to worry about all those magic IDs, CSS, or XPaths. Just tell the AI what you want it to find and it will find it for you on the page — even if the element changes color, text, location, or position in the DOM. With AI, tests will be a little quicker to create and will break a little less often. Welcome to the world of automation powered by AI!

一个API调用,你就能把人工智能的功能添加到你的手机自动化测试。 test.ai团队已经与 Jonathan Lipps在一起合作,他是Appium的领军贡献者和Cloud Grey的创始人,把一点人工智能添加到Appium。人工智能在手机应用查找通用元素就像在在输入框,登录按钮等,所以测试开发人员不需要担心所有这些神奇的IDs, CSS,或者XPath。只需要告诉人工智能你想要找到什么,它就会为你在页面找到——甚至元素改变了颜色,文本,位置,或者在DOM的位置。运用人工智能,测试在创建的时候速度会更快速,会更少中断操作。欢迎来到人工智能驱动的自动化测试。

How to Get Started 


How do you leverage the power of this AI brain in your code?



  1. Simply update your Appium project to the latest revision (see Jonathan Lipp’s Appium Pro article for the details)
  2. Then, find elements using a new custom AI-search strategy such as:

         // This code asks the AI to find a shopping cart image on the screen for you


  1. 您只需要升级您的Appium到最新的版本(详情可以查看Jonathan Lipp的文章:Appium Pro article
  2. :然后,用一个新的AI自定义搜索策略查找元素,比如:

// This code asks the AI to find a shopping cart image on the screen for you


I really can’t imagine it getting easier than this to add AI to your automation project. In fact, it is faster and simpler than the traditional methods of finding IDs, CSS or XPaths and using these more complicated search strategies.



AI For the Planet


Open source is key. No team should have to find the XPath or CSS Selector of an element or beg a developer to add a magic ID for them to use in their test code. No team should have to re-invent basic AI element classifiers or re-label 100,000+ images either — what a waste of humanity to duplicate that work. Therefore, the classifier is open source. Many vendors consider this type of IP their magic sauce, but that means that most test automation engineers can’t afford it, or don’t want to integrate it into their own code base. Open source means this tech is awesomely universally accessible to all.



Extensibility is key. This is the ‘hello world’ of bringing AI to element finding. Jonathan made sure this was a pluggable system, so any classifier can be used, or even other element search algorithms can be easily shared and plugged directly into Appium. Test.ai just open-sourced the default/reference implementation and donated to the community in the interest of sharing the power of AI with every test developer on the planet. Our mission at test.ai is to test the world’s apps. What better way is there than to help every test developer with the basics of finding elements inside of their own apps?

延展性也是重点。这是把人工智能带到元素查找的“世界”。Jonathan 确信,这是一个可插入的系统,所以任何分类器都能被用于,或者甚至其他的元素查找算法能被轻易的共享和直接插入到Appium。Test.ai 仅开源了默认值/参考实现,并且在共享人工智能驱动兴趣社区捐赠给地球上的每一个测试开发人员。我们在Test.ai 的使命是测试世界上的应用。还有什么比帮助每一个测试研发人员从基础定位他们自己APP内部元素更好的方式呢?


Customization is key. The testing community can improve the AI. Anyone can add new training data, alternative training methods, more rigorous relevance testing, or new labels. The AI is the property of the community, and we hope to help bootstrap every test team on the planet with a foundation of AI for their own projects. The test.ai team has shared all the training data on Kaggle, so the world can fork the data, clean it up, add it to their proprietary test frameworks, or compete to improve these classifiers. Crowdsourced, open data for AI testing systems? It is a new world.

定制化是关键。测试社区能够提高人工智能。任何人都能增加新的培训数据,可选的培训方法,更加严谨相关的测试,或者新标签。人工智能是社区的资产,我们希望能凭借我们人工智能的基础帮助地球上的每一个测试团队运用到他们的项目上。test.ai 团队已经在kaggle上分享了所有的培训数据,所以每个人都可以对数据进行分叉、清洗、加到自己的私有的测试框架,或者完成和改善这些分类器。为人工智能测试系统做众包或者开放数据?这是一个新的世界。


Reuse is key. The AI can be forked and/or re-used in other open source and proprietary frameworks. The goal at test.ai is to spread the usage of AI in all aspects of testing, in the interest of faster and smarter test automation, and ultimately better software in the world. The neural networks are based on the open source TensorFlow framework from Google. These models can be run in the cloud, locally, on mobile devices, or in a project not even thought of yet.

网络再利用关键。人工智能能被分叉和/或者在其他的开源软件或者私有化测试框架被重新使用。test.ai的目标是把人工智能推向测试的所有层面,在更快更智能的自动化测试领域,让世界的软件更好。神经网络就是基于谷歌的 TensorFlow框架。这些模型能够在运单或者本地或者手机设,甚至在任何一个未知的项目运行。


Become an AI Test Automation Engineer


Whether you are an AI expert, test automation geek, or just getting started with AI and testing, you can bring AI into your team and your project today. A free, open source, and single API call is all it takes. You can be the hero that brings a bit of AI to your engineering team. You can even contribute to this transformation in testing by helping add new training data or add a similar call to your favorite test framework — be an AI test automation engineer today. Ultimately, it will take our community to bring the power of AI to our entire field.



Seeing is believing and Jonathan Lipps created the first intro video demonstrating this working on both Android and iOS. What about web? It works there too, but it’s far less tested.

看到即相信,Jonathan Lipps创造看第一个介绍视频,这将能够在安卓和IOS运行。那web呢?它也能在这运行,但是远没有测试。


This is a hello world of real AI integrating with test automation tools to make our lives just a little easier, and hopefully more fun.



By the way, it is awesome working with Jonathan Lipps — truly today’s top expert in mobile test automation. Thanks to the team of machine learning and integration engineers at test.ai for the willingness and bravery to open source something you have poured so much energy and talent into the past year. And, special thanks to our investors who thought this was a great idea when I brought it up.

顺便说一下,跟Jonathan Lipps(当今在手机自动化测试的专家)一起工作太棒了。感谢 test.ai 在机器学习和集成集成工程师的团队,在过去的一年里你们为了开源软件的意愿和意志,倾注了如此巨大的能量和天赋。在此,特别感谢那些当初我带来这个想法时认为这是一个伟大想法的投资人。


“That’s one small step for AI, one giant leap for test automation” .


— Jason Arbon CEO, test.ai