Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32097

Allow reading history log files from multiple directories

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Wish
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 2.4.5
    • None
    • Spark Core
    • None

    Description

      Our service dynamically creates short-lived YARN clusters in cloud. Spark applications run on these dynamically created clusters. Log data for these applications is stored on a remote file-system. We want a static instance of SparkHistoryServer to view information on jobs that ran on these clusters. We use glob because we cannot have a static list of directories where the log files reside. 

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            gaurangi Gaurangi Saxena

            Dates

              Created:
              Updated:

              Slack

                Issue deployment