Date
Attendees
Saurabh
- Anu
- Marton
- Xiaoyu
- Nanda
- Sandeep
- Bharat
- Attilla
- Karthik
- Shewata
- Vivek
- Prasanth
- Mukul
Discussion items
| # | Item | Who | Notes |
|---|---|---|---|
| 1 | Ozone documentation is missing information about HDFS site dependencies | Saurabh |
|
| 2 | Data node missing UI | Saurabh | The Data node UI is missing – so unlike HDFS it is hard to understand what is happening. We discussed the Recon and the UI. We discussed the Namenode Analytics and how the design of Recon is based on that. |
| 3 | Recon and Security | Saurabh | Saurabh pointed out that he is not able to run NNA in financial institution due to legal and compliance requirements. He said that if Recon as the same level of Kerberos Support as OzoneManager and SCM, this should not be an issue. |
| 4 | Running Hive TPC-DS tests with Ozone | Saurabh | Saurabh was trying to run Hive based TPC-DS tests using Ozone. Cloudera has done this internally and (Pifta, not on call, shared this info with me). This needs some fixes in the current master branch. We also discussed the internal results that we have observed while testing Hive TPC-DS with Ozone. From Pifta:
|
| 5 | Ozone Cluster | Saurabh | He also mentioned that he has been running Ozone latest(0.4.1) for over 22 days without any issues. |
| 6 | Apache Release | Marton | Marton had proposed that we should do a follow up Apache release as soon as we do a Cloudera based release. Since there will be lot of testing cycles spend on Ozone during that time. Since TPC-DS runs need these fixes in the Master, we have decided to start a Discuss thread on Ozone mailing list for the 0.4.2-Alpha(Bryce) release. This will also be useful for anyone who is trying to run Ozone. |
| 7 | Basic Ozone File System Features | Saurabh/Xiaoyu/Marton | Saurabh mentioned that his company has not plans to upgrade to Hadoop 3.0 any time soon. The Ozone File system proper has dependencies on Hadoop 3.x. We also have a "basic" version of Ozone File system that can be used with Hadoop 2.7x. Saurabh wanted to get security and TDE support in this basic version of the Ozone FS driver. @Xiaoyu Yao, Seemed to think this can be done with Basic version of File system. |
| 8 | HBase Support in Ozone | Karthik | What is the status of running HBase on Ozone? or in other words, are we ready to go beyond Hive and Spark in our testing? Currently, we have very minimal support for HFlush. Some of the Hbase committers also expressed the interest in running HBase over Ozone. The plan of record is to finish the HFlush support and get Ozone working on it. Anu Engineer To follow up with Karthik and Josh Elser to see how we can make progress on this work item. |
| 9 | Du and other Metadata operations against Ozone | Karthik | The question was, what happens if we run a "Du" like operation against Ozone. Does it slow down the Ozone Manager?, the answer was that all such operations like Du and other management queries will be send to Recon. Recon looks and acts like SCM and OM and we will have near-real time answers for all the management queries. Since these requests are handled by an independent server, Ozone Manager should not be impacted by any query run by the admins of the system. |
| 10 | Tour of Sonar | Marton/Attilla/Vivek/Anu | Vivek, Marton and Attilla has enabled the SonarCloud support for Ozone. We have also fixed quite a lot of bugs in the system. Anu demoed the Sonar Cloud UI and explained how to use it spot bugs and fix things. The code coverage is 40% since we have enabled code coverage collection only on Unit tests. That is, integration and robot based tests are not used in code coverage numbers yet. Marton suggested that perhaps this is a good opportunity for us to write more unit tests. So we have proper coverage just by running unit tests. We often commit patches when unit tests pass, but if we have only 40% code coverage with unit tests, it is certainly a great idea to improve the code coverage. |