Fivetran & Managed Integration Solutions vs Manual Creation

By Tom Kozlowski | @intelia | June 30

What is Data Integration?

Data integration as defined in Gartner Information Technology Glossary:

“The discipline of data integration comprises the practices, architectural techniques and tools for achieving the consistent access and delivery of data across the spectrum of data subject areas and data structure types in the enterprise to meet the data consumption requirements of all applications and business processes”.

In other words, data integration is the process of combining data from various sources into a single, unified view for efficient data management. Data integration enables analytics tools to derive meaningful insights and gain actionable intelligence.

 

Challenges of Creating Your Own Data Integration

Creating their own data integration solution can give organisations flexibility over the design and implementation of data pipelines. However, many organisations utilise multiple applications and systems for their daily operations, often spanning across multiple data centres and cloud providers. Development of own data integration capability in such environments creates a number of challenges:

    • Lack of technical skills required to build data integration processes.
    • Lengthy and costly development effort is required to develop complex data integration solutions.
    • Ongoing maintenance of infrastructure and data integration processes.
    • Data engineers are diverted from performing higher-value tasks.

In some cases, where organisations use legacy or custom-built systems, it may be necessary to build data integration processes in-house. Managed data integration services may not be able to provide appropriate connectors due to the specificity or complexity of such systems. It is recommended to adopt a hybrid approach to extract data from such systems, where initial steps are developed in-house and extracted data is stored in a staging/temporary area accessible by managed data services.

Benefits of Managed Data Integration

Managed data integration services can include multiple components including application, data, cloud, and API integration. Outsourcing the management of data integration processes to an external service provider can have several benefits:

        • Expertise of professionals specialising in integration technology and processes to reduce effort/cost required to implement effective data integration solutions
        • Infrastructure management, monitoring and maintenance to ensure efficient operations and improved scalability of data integration solutions
        • Service providers can offer higher levels of data encryption, access controls, intrusion detection and compliance with relevant regulations and standards such as GDPR, HIPPA, etc
        • Unified and consistent user interface for all data sources
        • Built-in scheduling and automation capability
        • Service providers can offer data augmentation capability to improve quality of data.
        • Organisations can allocate their staff to more critical or higher value tasks to increase productivity
        • Organisations can achieve better integration between systems, applications and data sources, leading to improved business outcomes

While buying a managed solution may require sacrificing some bespoke configurations, it is cost effective and more efficient to implement and maintain. Managed solutions free up data engineers to explore more strategic options to improve business insights and ROI. Organisations will also benefit from the expertise and ongoing support of the vendors.

 

Fivetran Solution

Fivetran is the automated data movement platform moving data out of, into and across data centres and cloud data platforms. It offers data extraction and load capability from 300+ sources. With a fully managed data pipeline solution, Fivetran takes care of the most resource-intensive part of the process — writing code to move data. Organisations only determine what to buy and then configure it to suit their needs. Fivetran uses a consumption-based pricing model, where your cost is based on your use.

 

Fivetran offers secure database replication for every architecture – in the cloud for a fully managed experience, on-premises to meet even the strictest of data security requirements or hybrid for the best of both architectures:

  • Fully managed – cloud-hosted for a fully managed and automated data movement experience – choose the cloud service provider and region
  • Self-hosted – prevent data from leaving the organisation’s environment by deploying in VPC or on-premises to meet the strictest data movement requirements
  • Hybrid – control when and how data moves from the organisation’s on-premises environment to the cloud for efficient and secure database replication

Data Movement

Connect data sources and move data to target destinations with automated, reliable and scalable data movement platform:

  • 300+ pre-built, no-code source connectors
  • Automated schema drift handling, updates, data normalization and more
  • Performant change data capture for efficient and low impact data movement

Data Transformation

Accelerate time to insight with automated transformations, including:

  • Quickstart data models to complement the connectors, no dbt project needed
  • Integration, management and orchestration for custom dbt Core models
  • Lineage graphs to monitor end-to-end data movement

Data Security

Protect customers data and reputation with automated and customizable security features, including:

  • CCPA, GDPR, HIPAA, ISO, PCI, SOC2 certifications
  • Column blocking and hashing
  • Secure deployment and flexible connection option

Data Governance

Protect data in-flight from source to destination with automated governed data movement to support data democratization and self-service analytics:

  • Metadata sharing with data catalogue integrations
  • Advanced role-based access control
  • Automated user provisioning

Platform Integration and Extensibility

Leverage and extend the Fivetran platform to save development time and build better products with features including:

    • Workflow automation
    • Programmatically scale via the Fivetran API
    • Track data movement

Please contact us at contact@intelia.com.au should you have any issues with integrating data and want to try out a managed solution.