Hacky way to keep config files in sync when there’s a new version of some software. Caveat : probably completely wrong, may not pick up config entries added in the new version, etc. But, works for me right here right now ;-) So let’s say we have two folders: confluent-4.1.0 confluent-4.1.1 Same structures, different versions. 4.1.0 was set up with our local config in ./etc, that we want to preserve. We can use diff to easily see what’s changed:

Continue Reading

The Kafka Connect JDBC Connector by default does not cope so well with: NUMBER columns with no defined precision/scale. You may end up with apparent junk (bytes) in the output, or just errors. TIMESTAMP WITH LOCAL TIME ZONE. Throws JDBC type -102 not currently supported warning in the log. Read more about NUMBER data type in the Oracle docs. tl;dr : How do I make it work? There are several options:

Continue Reading

(preserving this StackOverflow answer for posterity and future Googlers) tl;dr When doing a stream-table join, your table messages must already exist (and must be timestamped) before the stream messages. If you re-emit your source stream messages, after the table topic is populated, the join will succeed. Example data Use kafakcat to populate topics: kafkacat -b localhost:9092 -P -t sessionDetails <<EOF {"Media":"Foo","SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1} {"Media":"Foo","SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2} EOF kafkacat -b localhost:9092 -P -t voipDetails <<EOF {"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1a"} {"SessionIdTime":"2018-05-17 11:25:33 BST","SessionIdSeq":1,"Details":"Bar1b"} {"SessionIdTime":"2018-05-17 11:26:33 BST","SessionIdSeq":2,"Details":"Bar2"} EOF Validate topic contents:

Continue Reading

tl;dr Use curl to pull data from the Mockaroo REST endpoint, and pipe it into kafkacat, thus:

curl -s "https://api.mockaroo.com/api/d5a195e0?count=2&key=ff7856d0"| \
kafkacat -b localhost:9092 -t purchases -P

Continue Reading

So the last post here was in 2011…seven years later I should probably post again, just to point random Google visitors to : My new blog : https://rmoff.net (Wordpress is icky; Ghost FTW!) My employer’s blog on which I also write: http://cnfl.io/rmoff

Continue Reading

Disclaimer: I am not a MongoDB person. These steps may or may not be appropriate and proper. But they worked for me :) Feel free to post in comments if I’m doing something wrong MongoDB config - enabling replica sets For Debezium to be able to stream changes from MongoDB, Mongo needs to have replication configured: Docs: Replication / Convert a Standalone to a Replica Set Stop Mongo: rmoff@proxmox01 ~> sudo service mongod stop Add replica set config to /etc/mongod.

Continue Reading

DISCLAIMER: I am not a MongoDB person (even if it is Web Scale X-D) - below instructions may work for you, they may not. Use with care! For some work I’ve been doing I wanted to access the data in Ubiquiti’s Unifi controller which it stores in MongoDB. Because I didn’t want to risk my actual Unifi device by changing local settings to enable remote access, and also because the version of MongoDB on it is older than ideal, I wanted to clone the data elsewhere.

Continue Reading

Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see how to set it up and examine the format of the data. A subsequent article will show using this realtime stream of data from a RDBMS and join it to data originating from other sources, using KSQL. The software versions used here are: Confluent Platform 4.0 Debezium 0.

Continue Reading

io.confluent.ksql.exception.KafkaTopicException: Topic 'KSQL_NOTIFY' does not conform to the requirements Partitions:1 v 4. Replication: 1 v 1 Why? Because the topic KSQL creates to underpin a CREATE STREAM AS SELECT or CREATE TABLE AS SELECT already exists, and doesn’t match what it expects. By default it will create partitions & replicas based on the same values of the input topic. Options: Use a different topic, via the WITH (KAFKA_TOPIC='FOO') syntax, e.g.

Continue Reading

This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard Oracle platform, and building streaming ETL using KSQL. This post shows how we take data streaming in from an Oracle transactional system into Kafka, and simply stream it onwards into Elasticsearch. This is a common pattern, for enabling rapid search or analytics against data held in systems elsewhere.

Continue Reading

System: rmoff@proxmox01:~$ uname -a Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux rmoff@proxmox01:~$ head -n1 /etc/os-release PRETTY_NAME="Debian GNU/Linux 8 (jessie)" rmoff@proxmox01:~$ python --version Python 2.7.9 Following: https://www.confluent.io/blog/introduction-to-apache-kafka-for-python-programmers/ https://github.com/confluentinc/confluent-kafka-python Install librdkafka, which is a pre-req for the Python library: wget -qO - https://packages.confluent.io/deb/4.0/archive.key | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.0 stable main" sudo apt-get install librdkafka-dev python-dev Setup virtualenv: sudo apt-get install virtualenv virtualenv kafka_push_notify source .

Continue Reading

(This is an expanded version of the intro to an article I posted over on the Confluent blog. Here I get to be as verbose as I like ;)) My first job from university was building a datawarehouse for a retailer in the UK. Back then, it was writing COBOL jobs to load tables in DB2. We waited for all the shops to close and do their end of day system processing, and send their data back to the central mainframe.

Continue Reading

This is the detailed step-by-step if you want to recreate the process I describe in the Confluent blog here I used Oracle’s Oracle Developer Days VM, which comes preinstalled with Oracle 12cR2. You can see the notes on how to do this here. These notes take you through installing and configuring: Swingbench, to create a sample “Order Entry” schema and simulate events on the Oracle database Oracle GoldenGate (OGG, forthwith) and Oracle GoldenGate for Big Data (OGG-BD, forthwith) I’m using Oracle GoldenGate 12.

Continue Reading

See also Kafka Listeners - Explained A short post to help Googlers. On a single-node sandbox Apache Kafka / Confluent Platform installation, I was getting this error from Schema Registry, Connect, etc: WARN [AdminClient clientId=adminclient-3] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) KSQL was throwing a similar error: KSQL cannot initialize AdminCLient. I had correctly set the machine’s hostname in my Kafka server.

Continue Reading

Some notes that I made on installing and configuring Oracle GoldenGate with Confluent Platform. Excuse the brevity, but hopefully useful to share! I used the Oracle Developer Days VM for this - it’s preinstalled with Oracle 12cR2. Big Data Lite is nice but currently has an older version of GoldenGate. Login to the VM (oracle/oracle) and then install some useful things: sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum install -y screen htop collectl rlwrap p7zip unzip sysstat perf iotop sudo su - cd /etc/yum.

Continue Reading

Here’s where I’ll be! If you use Google Calendar you can click on individual entries above and select copy to my calendar - which of course you’ll want to do for all the ones I’ve marked as [SPEAKING] :-) Here’s a list of all the Apache Kafka talks at OpenWorld and JavaOne, most of which I’ll be trying to get to.

Continue Reading

There’s an impressive 19 sessions that cover Apache Kafka™ at Oracle OpenWorld, JavaOne, and Oak Table World this year! You can find the full list with speakers in the session catalogs for OOW, JavaOne, and Oak Table World. OTW is an awesome techie conference which is at the same time as OpenWorld, next door to Moscone. Hope to see you there! Check out the writeup of my previous visit to OOW including useful tips here.

Continue Reading

The Replicat was kapput: GGSCI (localhost.localdomain) 3> info rkconnoe REPLICAT RKCONNOE Last Started 2017-09-12 17:06 Status ABENDED Checkpoint Lag 00:00:00 (updated 00:46:34 ago) Log Read Checkpoint File /u01/app/ogg/dirdat/oe000000 First Record RBA 0 So checking the OGG error log ggserr.log showed 2017-09-12T17:06:17.572-0400 ERROR OGG-15051 Oracle GoldenGate Delivery, rkconnoe.prm: Java or JNI exception: oracle.goldengate.util.GGException: Error detected handling operation added event. 2017-09-12T17:06:17.572-0400 ERROR OGG-01668 Oracle GoldenGate Delivery, rkconnoe.prm: PROCESS ABENDING. So checking the replicat log dirrpt/RKCONNOE_info_log4j.

Continue Reading

Markdown is a plain-text formatting syntax. It enables you write documents in plain text, readable by others in plain text, and optionally rendered into nicely formatted PDF, HTML, DOCX etc. It’s used widely in software documentation, particularly open-source, because it enables richer formatting than plain-text alone, but without constraining authors or readers to a given software platform. Platforms such as github natively support Markdown rendering - so you write your README etc in markdown, and when viewed on github it is automagically rendered - without you needing to actually do anything.

Continue Reading

I’m excited to be speaking at several conferences and meetups over the next few months. Unsurprisingly, the topic will be Apache Kafka! If you’re at any of these, please do come and say hi :) Apache Kafka Meetup - London My first time talking at the London Apache Kafka Meetup - always a sold-out crowd, this will be fun! September 20th, 19:00 : Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka Slides are available here Oracle OpenWorld - San Francisco This will be my second time at OOW - I wrote up my previous trip here

Continue Reading