Introducing Quickbase Pipelines?
Pipelines, a relatively new addition to Quickbase, provide advanced capabilities for workflow, process automation, and integration, giving companies functionality on-par with leading integration platform-as-a-service (iPaaS) solutions. Quickbase account holders get capabilities similar to Workato, Zapier, or Microsoft Flow without requiring additional technologies or licenses.
Pipelines offer a perfect complement to Quickbase because, in the same way that Quickbase allows fairly non-technical users to rapidly take a business concept, process, or requirement and turn it into a functional software solution with little or no code, Pipelines simply require configuration instead of development. Pipelines work with the same rapid development paradigm Quickbase with a focus on workflow automation and integration.
What can Quickbase users do with Pipelines?
The principal use case of Pipelines centers around extracting data from a system outside of Quickbase and loading it to a Quickbase application or vice versa without requiring complex development. However, neither the source nor destination systems need to be Quickbase, so Pipelines can serve as a standalone iPaaS solution to integrate and automate interactions between pretty much any different systems that can be connected.
Pipelines also provide powerful capabilities for handling ETL (extract transform and load) operations, where data from one system needs to be substantially modified before loading to another. For example, if extracting some data like time-cards from QuickBooks Online to load to Quickbase requires manipulation of the data to properly match the target data or enhance the business process, Pipelines enables features to easily handle the data transformation. So not only am I pulling time cards from QuickBooks online and pulling them into Quickbase. Transformations can be anything from switching people from last name, comma first name to first name, last name with no comma, or converting currency, or more complex operations.
Either to support process automation, integration or just to do things like data cleanup, or manipulation and provide analytics on data, Pipelines can support a wide variety of needs that substantially enhance the power of Quickbase.
Example use cases for Quickbase Pipelines
Outlook Calendar Integration
A common use case is to leverage the Outlook Channel to connect to a specific Calendar. For example, someone may be using Quickbase as a CRM, wants to schedule a follow-up meeting with a prospect and capture the activity without double entering data. Pipelines allows the record to be entered in Quickbase, associated with a date, time, and a person, and then automatically generates the calendar event and invitation in Outlook.
Outlook Email Integration
Another great use of the Pipelines Outlook Channel uses the Email connection to ingest messages into Quickbase. Perhaps a business is using Quickbase as an IT or technical support team to manage help tickets and want to support Ticket submission via email. They can set up a Support email and easily automate Ticket creation from emails sent in. An extension of this might be to use the Slack Channel and automatically post a message to the Support Team alerting them when any new help request comes into queue.
Slack Channel Integration
Pipelines come with a Slack Channel out of the box, which can be very useful to compliment many processes. Slack has become very common with Corporate and Small Business teams, providing a wide range of benefits, though it can sometimes create a sense of operating outside of the business information systems. To make Slack more complimentary to structured tools, integrating data, process, and messaging can tie everything together. For example, a step can be added into a workflow to notify certain people or teams via Slack and provide some essential information in the message. It’s also possible to drive a Slack Bot to take certain specific actions and even capture some of the dialogue back into Quickbase. In this way, Slack can be more effective and operate as a centralized service for ad hoc chatting as well as alerts about relevant business process updates.
Quickbase Pipelines Basics
Getting Started
The Pipelines feature of Quickbase operates like an independent application (it’s not a feature inside one of your existing applications) and can be found under My Apps where users will see Pipelines as an application you can select to access.
When users are ready to start using Pipelines, they can just access it from the Menu and dive in.
Pipeline Configuration
Before starting to build Pipelines it’s necessary to set up some channels which will be needed to connect to various data or services. When adding the Channels users need to authenticate with the external service and then they will be available to use them in any of their Pipelines.
Once the desired Channels have been enabled, from the Pipelines application users can simply create a Pipeline to begin configuration. A key initial decision will be to decide whether the Pipeline will be triggered or scheduled. Triggered Pipelines will fire when a certain event occurs in Quickbase or one of your Channels, such as an email arriving or a change in some data. A scheduled PipeIine will fire at a specified interval of time, such as hourly, daily, etc.
In the case of a triggered event in an Outlook Channel, users are able to select the specific trigger from a list of possible triggers for the channel and then complete any specifics required to detail the event.
After configuring the trigger, users then need to specify what will happen with the data received from the source system, in terms of how the data might be manipulated (more details on this later) and where it will be sent by adding steps in the Pipeline.
Deploying Pipelines
It’s important to understand that Pipelines are currently tied to a specific Quickbase user as the Pipeline owner so there isn’t a current model for managing group access or control for deployment. Administrators are able to see an overview of Pipelines running and perform some basic control features.
On the flip side, testing of Pipelines provides robust features to step through each stage of the Pipeline and see the input and output of the stage. This makes it easy to see what data is being received from a source system, how that data appears after any transformation, and what data is being sent to any target systems. For example, when loading in an email users can see the raw subject, body, to, from, CC, etc. Once that data is pushed to Quickbase the API response can be viewed to verify the record was properly added or address any errors, right down to the field to field mapping.
Managing & Monitoring Pipelines
The Pipelines application provides a dashboard upon accessing which shows all of the currently operating Pipelines with real-time updates when triggers fire and an activity feed showing recent actions. For example, if a QuickBooks job just initiated users will see an alert like: “QuickBooks invoice sync just started,” and users can essentially watch the process run in real-time.
In addition, each Pipeline has its own Dashboard with an activity log that users can drill into, see the history of execution and even drill further down into the individual inputs and outputs of every single step. This level of logging and monitoring on each Pipeline provides a powerful feature for checking on the stability of the processes and troubleshooting if issues occur.
The main Pipelines dashboard and each individual Pipeline dashboard also provide details on the metering related to account billing as Pipelines are charged in Quickbase on a consumption basis.
Insider Tips
Using Jinja
Many Pipelines can be built without requiring much technical experience. However, more complex ETL transformations and logical operations may require some more advanced skill. Pipelines use a Python-based syntax for manipulating data called Jinja. Jinja supports simple yet powerful capabilities to operate on source data and transform using calculations, logical statements, and even summaries.
For example, a fairly non-technical user could quickly grab emails and send them to Quickbase or Slack. However, the email content might have some unwanted content such as HTML tags, etc. which might require some manipulation with Jinja to strip out the junk.
Jinja operates similarly to SQL and can offer very powerful means to search datasets, query, summarize, aggregate, and more.
Getting familiar with Jinja can unlock some pretty amazing opportunities with Pipelines.
https://jinja.palletsprojects.com/en/2.11.x/
Using REST API’s
While pre-built Channels will help users connect to many popular applications, numerous others will need to be accessed via the Webhook Channel which requires some understanding of how to authenticate and operate with Rest APIs.
Most SaaS and internet applications provide Rest APIs as a means to integrate with the data. However, the methods for authenticating, querying, and posting data can vary substantially. Studying the source system API’s and getting familiar with the common approaches to Rest API’s will likely be valuable time spent for users developing a variety of Pipelines.
https://tray.io/blog/what-is-an-api-integration-for-non-technical-people
Using a Quickbase Partner for Help
With the low/no-code promise and power of Quickbase, many users will develop powerful applications with little or no professional assistance. As applications become more mission-critical and requirements more complex, professional services from a Quickbase partner may be the difference in achieving long term success.
A certified Quickbase partner such as MCFTech can help implement advanced Pipelines with complex ETL transformations, help integrate with systems where out-of-the-box Channels aren’t available by leveraging extensive API experience, and design automations based on best practices from many years helping clients succeed with Quickbase.
In addition, MCFTech can assist with new advances as Pipelines evolve such as the upcoming release of an ODBC channel and onsite client, which will securely allow Pipeline integration with on-premise Corporate data in SQL and other legacy data systems.
Contact us more to learn how we can help automate your business.