http://wiki.old.lustre.org/index.php?action=history&feed=atom&title=Running_Hadoop_with_Lustre 2026-04-17T18:53:25Z Revision history for this page on the wiki MediaWiki 1.39.7 http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=11229&oldid=prev 2010-02-22T19:00:06Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 12:00, 22 February 2010</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1">Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">&lt;small>''(Updated: Jan 2010)''&lt;/small></ins></div></td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This project was conducted by three interns as a proof of concept study for how to integrate Apache Hadoop with Lustre™. Although the performance tests, conducted by the interns to compare Lustre to HDFS when used with Hadoop, did not show a significant performance benefit for Lustre, this study has laid the groundwork for further work in this area.  </div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>This project was conducted by three interns as a proof of concept study for how to integrate Apache Hadoop with Lustre™. Although the performance tests, conducted by the interns to compare Lustre to HDFS when used with Hadoop, did not show a significant performance benefit for Lustre, this study has laid the groundwork for further work in this area.  </div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l26">Line 26:</td> <td colspan="2" class="diff-lineno">Line 28:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>The paper [[Media:Hadoop_wp_v0.4.2.pdf|&#039;&#039;Using Lustre with Apache Hadoop&#039;&#039;]] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>The paper [[Media:Hadoop_wp_v0.4.2.pdf|&#039;&#039;Using Lustre with Apache Hadoop&#039;&#039;]] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"></del></div></td><td colspan="2" class="diff-side-added"></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">''(Updated 01/10)''</del></div></td><td colspan="2" class="diff-side-added"></td></tr> </table>

Sbarthel http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=11003&oldid=prev 2010-02-18T21:40:13Z

<p><span dir="auto"><span class="autocomment">Comparing the Performance of Lustre and HDFS</span></span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 14:40, 18 February 2010</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l26">Line 26:</td> <td colspan="2" class="diff-lineno">Line 26:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>The paper [[Media:Hadoop_wp_v0.4.2.pdf|&#039;&#039;Using Lustre with Apache Hadoop&#039;&#039;]] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>The paper [[Media:Hadoop_wp_v0.4.2.pdf|&#039;&#039;Using Lustre with Apache Hadoop&#039;&#039;]] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">''(Updated 01/10)''</ins></div></td></tr> </table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=9552&oldid=prev 2010-01-09T02:13:00Z

<p>moved <a href="/index.php?title=Integrating_Hadoop_with_Lustre&amp;action=edit&amp;redlink=1" class="new" title="Integrating Hadoop with Lustre (page does not exist)">Integrating Hadoop with Lustre</a> to <a href="/index.php?title=Running_Hadoop_with_Lustre" title="Running Hadoop with Lustre">Running Hadoop with Lustre</a></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <tr class="diff-title" lang="en"> <td colspan="1" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="1" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 19:13, 8 January 2010</td> </tr><tr><td colspan="2" class="diff-notice" lang="en"><div class="mw-diff-empty">(No difference)</div> </td></tr></table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=9551&oldid=prev 2010-01-09T02:07:45Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 19:07, 8 January 2010</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1">Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>This <del style="font-weight: bold; text-decoration: none;">page provides access to information about </del>how to integrate Apache Hadoop with Lustre™. <del style="font-weight: bold; text-decoration: none;">We have made several enhancements to improve </del>the <del style="font-weight: bold; text-decoration: none;">use of Hadoop with Lustre and have conducted </del>performance tests to compare <del style="font-weight: bold; text-decoration: none;">the performance of </del>Lustre <del style="font-weight: bold; text-decoration: none;">vs. </del>HDFS when used with Hadoop.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>This <ins style="font-weight: bold; text-decoration: none;">project was conducted by three interns as a proof of concept study for </ins>how to integrate Apache Hadoop with Lustre™. <ins style="font-weight: bold; text-decoration: none;">Although </ins>the performance tests<ins style="font-weight: bold; text-decoration: none;">, conducted by the interns </ins>to compare Lustre <ins style="font-weight: bold; text-decoration: none;">to </ins>HDFS when used with Hadoop<ins style="font-weight: bold; text-decoration: none;">, did not show a significant performance benefit for Lustre, this study has laid the groundwork for further work in this area</ins>.  </div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>== <del style="font-weight: bold; text-decoration: none;">Advantages </del>of Using Hadoop with Lustre ==</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>== <ins style="font-weight: bold; text-decoration: none;">Benefits </ins>of Using Hadoop with Lustre ==</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Using Hadoop with Lustre provides several <del style="font-weight: bold; text-decoration: none;">advantages</del>, including:</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Using Hadoop with Lustre provides several <ins style="font-weight: bold; text-decoration: none;">benefits</ins>, including:</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Lustre is a real parallel file system, which enables temporary or intermediate data to be stored in parallel on multiple nodes reducing the load on single nodes.  </div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Lustre is a real parallel file system, which enables temporary or intermediate data to be stored in parallel on multiple nodes reducing the load on single nodes.  </div></td></tr> <tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l11">Line 11:</td> <td colspan="2" class="diff-lineno">Line 11:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Lustre can be mounted as a standard POSIX file system.</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Lustre can be mounted as a standard POSIX file system.</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>== <del style="font-weight: bold; text-decoration: none;">Disadvantages </del>of Using Hadoop with HDFS ==</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>== <ins style="font-weight: bold; text-decoration: none;">Drawbacks </ins>of Using Hadoop with HDFS ==</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Using Hadoop with HDFS has several drawbacks, including:</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Using Hadoop with HDFS has several drawbacks, including:</div></td></tr> </table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=9464&oldid=prev 2009-12-19T20:27:10Z

<p><span dir="auto"><span class="autocomment">Comparing the Performance of Lustre and HDFS</span></span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 13:27, 19 December 2009</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l25">Line 25:</td> <td colspan="2" class="diff-lineno">Line 25:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Comparing the Performance of Lustre and HDFS ==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Comparing the Performance of Lustre and HDFS ==</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>The paper [<del style="font-weight: bold; text-decoration: none;">http</del>:<del style="font-weight: bold; text-decoration: none;">//wiki.lustre.org/images/1/1b/</del>Hadoop_wp_v0.4.2.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>The paper [<ins style="font-weight: bold; text-decoration: none;">[Media</ins>:Hadoop_wp_v0.4.2.pdf<ins style="font-weight: bold; text-decoration: none;">|</ins>&#039;&#039;Using Lustre with Apache Hadoop&#039;&#039;<ins style="font-weight: bold; text-decoration: none;">]</ins>] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div> </div></td><td colspan="2" class="diff-side-added"></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">[[Media:Hadoop_wp_v0.4.2.pdf|&#039;&#039;Using Lustre with Apache Hadoop&#039;&#039;]]</del></div></td><td colspan="2" class="diff-side-added"></td></tr> </table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=9401&oldid=prev 2009-12-19T00:36:05Z

<p><span dir="auto"><span class="autocomment">Comparing the Performance of Lustre and HDFS</span></span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 17:36, 18 December 2009</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l26">Line 26:</td> <td colspan="2" class="diff-lineno">Line 26:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>The paper [http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>The paper [http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">[[Media:Hadoop_wp_v0.4.2.pdf|''Using Lustre with Apache Hadoop'']]</ins></div></td></tr> </table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=9143&oldid=prev 2009-12-16T18:08:34Z

<p>Protected &quot;<a href="/index.php?title=Integrating_Hadoop_with_Lustre&amp;action=edit&amp;redlink=1" class="new" title="Integrating Hadoop with Lustre (page does not exist)">Integrating Hadoop with Lustre</a>&quot; ([edit=sysop] (indefinite) [move=sysop] (indefinite))</p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <tr class="diff-title" lang="en"> <td colspan="1" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="1" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 11:08, 16 December 2009</td> </tr><tr><td colspan="2" class="diff-notice" lang="en"><div class="mw-diff-empty">(No difference)</div> </td></tr></table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=9142&oldid=prev 2009-12-16T18:08:24Z

<p></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 11:08, 16 December 2009</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1">Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>This page provides access to information about how to integrate Apache Hadoop with <del style="font-weight: bold; text-decoration: none;">Lustre</del>. We have made several enhancements to improve the use of Hadoop with Lustre and have conducted performance tests to compare the performance of Lustre vs. HDFS when used with Hadoop.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>This page provides access to information about how to integrate Apache Hadoop with <ins style="font-weight: bold; text-decoration: none;">Lustre™</ins>. We have made several enhancements to improve the use of Hadoop with Lustre and have conducted performance tests to compare the performance of Lustre vs. HDFS when used with Hadoop.</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Advantages of Using Hadoop with Lustre ==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Advantages of Using Hadoop with Lustre ==</div></td></tr> </table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=6490&oldid=prev 2009-08-06T18:07:15Z

<p><span dir="auto"><span class="autocomment">Comparing the Performance of Lustre and HDFS</span></span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 11:07, 6 August 2009</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l25">Line 25:</td> <td colspan="2" class="diff-lineno">Line 25:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Comparing the Performance of Lustre and HDFS ==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Comparing the Performance of Lustre and HDFS ==</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>The paper [http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.<del style="font-weight: bold; text-decoration: none;">2_</del>.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>The paper [http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.<ins style="font-weight: bold; text-decoration: none;">2</ins>.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td></tr> </table>

Docadmin http://wiki.old.lustre.org/index.php?title=Running_Hadoop_with_Lustre&diff=6489&oldid=prev 2009-08-06T18:06:40Z

<p><span dir="auto"><span class="autocomment">Comparing the Performance of Lustre and HDFS</span></span></p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 11:06, 6 August 2009</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l25">Line 25:</td> <td colspan="2" class="diff-lineno">Line 25:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Comparing the Performance of Lustre and HDFS ==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Comparing the Performance of Lustre and HDFS ==</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>The paper [http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.<del style="font-weight: bold; text-decoration: none;">2</del>.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>The paper [http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.<ins style="font-weight: bold; text-decoration: none;">2_</ins>.pdf&#039;&#039; Using Lustre with Apache Hadoop&#039;&#039;] provides an overview of Hadoop and HDFS and describes how to set up Lustre with Hadoop. It also provides performance results for several categories of test cases including application tests (word counts, reading and outputting large non-splittable files), sub-phase tests to identify the time-consuming phases for specific applications, and benchmark I/O tests.</div></td></tr> </table>

Docadmin