Flink Live Shopper Features with Redis

A Snowplow solution accelerator showcasing real-time user behavior analytics using Snowplow event tracking, processed by Apache Flink, and stored in Redis for potential downstream use or visualization.

Written in Java by our friends at Evoura.

Quickstart

To get started, head to our Real-time shopper features using Apache Flink: Introduction in the Snowplow Documentation site.

Technical details

Overview

The system captures user interactions (product views, cart actions, purchases, etc.) from a demo Next.js e-commerce store using the Snowplow JavaScript tracker. These events are sent to a Snowplow collector, ingested into Kafka, and then processed in real-time by an Apache Flink pipeline. The Flink pipeline calculates various metrics and features based on user behavior within different time windows (rolling and session windows), including purchase history analysis. The results (metrics) are stored in Redis for potential downstream use or visualization.

Key technologies

• Snowplow: Event tracking pipeline (Collector, Enrich, Kafka sink)
• Apache Flink: Stream processing engine for real-time analytics
• Apache Kafka: Message broker for decoupling event producers and consumers
• Redis: In-memory data store for storing computed metrics
• Next.js: Framework for the demo e-commerce application
• Docker & Docker Compose: Containerization for easy setup and deployment of all services
• AKHQ: Web UI for Kafka management and inspection
• Redis Insight: Web UI for Redis data visualization and management
• Grafana: Visualization tool for monitoring and analyzing metrics

Project structure

• apps/: Contains the Flink processing application (flink/) and the demo e-commerce store (ecommerce/).
• infra/: Contains Docker Compose configurations for the infrastructure components (Snowplow, Kafka, Redis, Localstack).
• docker-compose.yaml: Main Docker Compose file including all services.
• up.sh: Convenience script to initialize and start all services.

Running the project

1. Prerequisites:

   • Docker and Docker Compose installed.
   • Git installed (for submodules).

2. Clone the Repository (if you haven't already):

   git clone <repository-url>
   cd <repository-directory>

3. Initialize and Start Services: Run the provided script. This will:

   • Ensure .env files exist (copying from examples if needed).
   • Update Git submodules.
   • Start all services defined in the Docker Compose files in detached mode.

   ./up.sh

4. Access Services: Once the script completes successfully, the services will be available at:

   • 🛒 Ecommerce Store: http://localhost:3000
   • 👁️ AKHQ (Kafka UI): http://localhost:8085
   • 📊 Redis Insight: http://localhost:5540
   • 📈 Grafana: http://localhost:3001
   • 📡 Snowplow Collector: Receives events at http://localhost:9090
   • 🛠️ Kafka Brokers: Accessible internally at kafka:9092 and externally at localhost:19092 (check infra/kafka/docker-compose.yaml for exact external port if needed).

5. Stopping Services:

   docker compose down

Development

• The Flink job (SnowplowAnalyticsPipeline.java) processes events based on different windowing strategies:
  • Rolling windows for product views, category interactions, purchase history, and cart behavior.
  • Session windows for overall session analysis.
• Metrics generated by Flink are stored as key-value pairs in Redis.
• Events can be monitored in Kafka using AKHQ.
• Redis data can be inspected using Redis Insight.

Special thanks

This solution accelerator was built by the team at Evoura, the data streaming & Apache Flink experts.

Project support

Please note the code in this project is provided for your exploration only, and are not formally supported by Snowplow with Service Level Agreements (SLAs). Solutions accelerators are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects. The source in this project is provided subject to the license set forth below. All included or referenced third party libraries are subject to the licenses set forth below.

Any issues discovered through the use of this project should be filed as GitHub Issues on the repo. They will be reviewed as time permits, but there are no formal SLAs for support.

Copyright and license

Flink Live Shopper Features with Redis is copyright 2025-present Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.