Testing and Interpretation

Neu.ro Tool Integrations

In our experience, nearly all AI development efforts, be they at large enterprises or new startups, begin by spending the first 3-6 months building their first ML pipelines from available tools. These custom integrations are time consuming and expensive to produce, can be fragile and frequently require drastic changes as project requirements evolve.

Frequently, these custom ML pipelines only support a small set of built-in algorithms or a single ML library and are tied to each company’s infrastructure. Users cannot easily leverage new ML libraries, or share their work with a wider community.

Neuro facilitates adoption of robust, adaptable Machine Learning Operations (MLOps) by simplifying resource orchestration, automation and instrumentation at all steps of ML system construction, including integration, testing, deployment, monitoring and infrastructure management.

To maintain agility and avoid the pitfalls of technical debt, Neuro allows for the seamless connection of an ever-expanding universe of ML tools into your workflow.

We cover the entire ML lifecycle from Data Collection to Testing and Interpretation. All resources, processes and permissions are managed through our Neu.ro platform and can be installed and run on virtually any compute infrastructure, be it on-premise or in the cloud of your choice.

Testing and Interpretation

The various components of a machine learning workflow can be split up into independent, reusable, modular parts that can be pipelined together to create, test and deploy models.

Our toolset integrator, Neu.ro Toolbox, contains up to date out of the box integrations with a wide range of open-source and commercial tools required for modern ML/AI development.

For Testing and Interpretation, Neu.ro provides out of the box integrations with Fiddler, Seldon and WhyLabs.

Fiddler:

Fiddler is a leading tool for model interpretation. Fiddler augments top AI explainability techniques in the public domain including Shapley Values and Integrated Gradients to enhance performance. This allows data scientists to get deep model-level actionable insights to understand problem drivers using explanations and efficiently root cause issues, and gives immediate visibility into performance issues before they can result in negative business impact.

Fiddler is used to efficiently solve specific operational challenges like drift, and outliers with always-on real-time explainable ML monitoring and allows users to bring in data and models from any platform to explain them in Fiddler using the best interpretability methods available – Shapley Values and Integrated Gradients – made fast, reliable and scalable.

Fiddler’s implementation of confidence intervals, which they describe as a significant upgrade to other Shapley Values implementations, is outlined in their paper ‘Explanation Game‘.

Fiddler further allows users to Compare distributions across training data, test data and production data, and to slice data into groups for explanation of performance discrepancies across groups.

Analyze your AI predictions in relation to the entire data set or just a specific region to find anomalies and drifting data.

Seldon

Seldon is a leading machine learning framework for the rapid deployment of machine learning models on Kubernetes and to manage, serve and scale models in any language or framework. Seldon simplifies the process of testing, monitoring and deploying models in live environments through intuitive dashboards and greater collaboration between data scientists and DevOps teams. Seldon also allows for integration with external continuous integration and deployment (CI/CD) tools to scale and update deployments and makes use of powerful Kubernetes features such as custom resource definitions to manage model graphs.

In neuro-extras
- Auxiliary scripts and recipes for automating routine tasks.
- set of tools and commands extending functionality of neu.ro CLI tool.
- Docs: https://neu-ro.gitbook.io/neuro-extras-reference/cli#neuro-extras-seldon
- Neuro-extras seldon generate-deployment
- Neuro-extras seldon init-package
- neuro-extras seldon generate-deployment

WhyLabs:

WhyLabs is a leading AI Observability solution that works on both structured and unstructured data and allows users to monitor raw data, feature data, predictions and actuals.

WhyLab is known for fast set up, and seamless integration with existing data pipelines and on-premises and multi-cloud architectures.The solution produces real-time actionable insights in minutes.

Real-time monitoring allows users to surface data drift, data bias, and data quality issues; monitor model accuracy and concept drift; ensure that models make recommendations accurately and consistently; and prevent forecasting bias caused by data quality and seasonal drifts;

WhyLabs increases ROI by continuously improving model performance across edge cases

Frequently, these custom ML pipelines only support a small set of built-in algorithms or a single ML library and are tied to each company’s existing infrastructure. Users cannot easily leverage new ML libraries, or share their work with a wider community.

To maintain agility and avoid the pitfalls of technical debt, Neuro allows for the seamless connection of an ever-expanding universe of ML tools into your workflow.

Deployment

The various components of a machine learning workflow can be split up into independent, reusable, modular parts that can be pipelined together to create, test and deploy models.

Our toolset integrator, Neu.ro Toolbox, contains up to date out of the box integrations with a wide range of open-source and commercial tools required for modern ML/AI development.

For Deployment, the Neu.ro Platform provides out of the box integrations with Algorithmia and Seldon.

Request A Demo