Using R Shiny to create a plot with a shapefile and a raster

Your code has several structural problems with the reactives and so on, and some possible more fundamental problems with shapefile.

Here's a working version with some caveats that follow:

ui=shinyUI(fluidPage(pageWithSidebar(
  headerPanel("Header1"),
  sidebarPanel(
      fileInput('layer', 'Choose Layer', multiple=FALSE, accept='asc'),
      fileInput('shape', 'Choose gml', multiple=FALSE, accept="gml")

  ),
  mainPanel(
      plotOutput("mapPlot")
  )
)))


server = shinyServer(function(input,output){

    inFile <- reactive({
        raster::brick(input$layer$datapath)
    })

    inShp = reactive({
        readOGR(input$shape$datapath)
    })

    output$mapPlot<-renderPlot(
    {
        plot(inFile());
        plot(inShp(), add=TRUE)
    })
})

Note that everything in the server function is either in a "reactive" or a "renderer". That's what was causing your initial problem.

Secondly the file upload input returns a data frame with a datapath column which is where the file has been dropped. So in your code you would have been trying to read from input$layer which is a dataframe of upload data rather than input$layer$datapath which is the column with the file name you want.

Thirdly it is hard to make shapefiles work with shiny uploads. Shapefiles must have a .shp and a .shx component for readOGR to work, and possibly a .dbf and .prj component too. You can upload several files in a shiny upload widget (is that why you have multiple=T?) but the server renames them so that the data frame of input$shape contains the names 0.shp, 1.shx, 2.dbf and so on. It might be possibly to rename these back together, but most shapefile upload solutions require users to upload a ZIP file of everything which is then extracted by the server.

My solution above, and in order to show how the reactive stuff works which was your first problem, is to load a GML file instead of a shapefile. In my test code I did writeOGR(pts, "pts.gml","pts", driver="GML") just to get something I could upload to shiny and plot over a raster.