Introduction
Recently, I’ve done load testing
of impala. During testing I’ve found by doing some tiny changes query
performance can be improve drastically. I would like to share my experience to
all. Majorly I’ve found you should tune impala by two ways.
Less your scanning time of HDFS(storage):
It’s always primary
concern to read data from distributed mode. Data scatter across the multiple machines.
You’re limited to disk throughput and network I/O. We’ve found scanning of HDFS
taking so long time then to do actual aggregation jobs. There are basically two reasons for taking long
time:
1) Scan data from disk:
Disk throughput
is limited to MBPS. In our testing I’ve found it goes to max 150mbps(on m3.Xlarge instance on aws EC2 and 10 IOPS). i.e. if
you are reading 50 GB data from scatter disk it will take around 6 minutes only
for reading your data and We expect query response in less than 3 to 5 sec in real time. I
would also like to point your direction on concurrent read. Reading tens of
concurrent user from disk can kill your system for endless time period. So does
it mean we can’t reduce the time?
Answer is no, we
can. Newer Cloudera distribution come up with the concept of HDFS caching. You
can cache your data into physical memory. I’ve seen the reading from cache data
gives you GBPS speed. It can drastically change scanning time of HDFS. And also
this reflection you can see on concurrent reads also.
2) Scan full data:
Scanning full data
is sometimes panic for system and that also if I consider it sequential
scanning. What if I interested only particular app id, your system will scan
complete data and filter your data row by row. What if I get something where I
get already filtered data there I don’t need to scan my complete data set. And
this mechanism called Partition. We can partition our data by some predefined
filters. So that impala will scan only those data set which you are put into
filter conditions.
Aggregate level
I’ve also
found you can tune impala by your query also. As for example we can do the date
parsing before giving the input to impala. So that Impala not take much time on
parsing and filtering the rows. Joining optimization can also be consider.
Summary:
Impala is a great
tool; If you optimize it from storage level to processing level. Storage and
processing equally put the impact on system. Sometimes, an optimization technique improves scalability more
than performance. For example, reducing memory usage for a query might not
change the query performance much, but might improve scalability by allowing
more Impala queries or other kinds of jobs to run at the same time without
running out of memory.
