Re: [问题] shiny读取大档案csv会当机

楼主: celestialgod (天)   2017-10-21 22:42:58
※ 引述《Esmelee (Esme)》之铭言:
: 文章分类提示:
: - 问题:shiny
: [问题类型]:
: 程式咨询(我想用R 做某件事情,但是我不知道要怎么用R 写出来)
: [软件熟悉度]:
: 入门(写过其他程式,只是对语法不熟悉)
: [问题叙述]:
: 我做了一个shiny读取历年资料的程式
: 可是我的csv档案有2gb大
: 发现读取可以,可是要用shiny+套件时就会当机
: 如果不用shiny就可以正常绘图了
: 有人建议我用mysql可是我把资料弄进mysql,mysql就当机了
: 档案太大吗??只有2g阿
: 到底要怎么用才能正常在shiny上跑呢
: 我的内存有16g
: 我是不是要用什么spark hadoop的就会跑得出来
: 而且后端档案有2g大网站要怎么做呢
: shinyapps.io好像不可能...
帮你做个简单的修改,只是改用data.table
因为data.frame在过程中会不断被复制,会增加内存使用量,拖慢速度
给个测试例子:https://pastebin.com/tHZ8UGpx
: [程式范例]:
: library(leaflet)
: library(shiny)
: library(shinydashboard)
: library(readr)
: library(methods)
: library(DT)
: library(RCurl)
library(data.table)
alldata_2010 <- fread("alldata_test.csv",
colClasses = c("PM2.5" = "numeric",
year = "character",
month = "character",
day = "character",
hour = "character"))
setkey(alldata_2010, year, month, day, hour)
: ui <-
: fluidPage(
: titlePanel("Basic DataTable"),
: fluidRow(
: column(4,
: selectInput("year",
: "year:", alldata_2010, selectize=TRUE)
: ),
: column(4,
: selectInput("month",
: "month:", alldata_2010, selectize=TRUE)
: ),
: column(4,
: selectInput("day",
: "day:", alldata_2010, selectize=TRUE)
: ),
: column(4,
: selectInput("hour",
: "hour:", alldata_2010, selectize=TRUE)
: ),
: fluidRow(
: title = "data MAP",
: collapsible = TRUE,
: width = "100%",
: height = "100%",
: leafletOutput("datamap", height = "900px")
: )
: )
: )
: server <-
: function(input, output) {
: output$datamap <- renderLeaflet({
data <- copy(alldata_2010)
: if (input$year != "All") {
data <- data[year == input$year]
: }
: if (input$month != "All") {
data <- data[month == input$month]
: }
: if (input$day != "All") {
data <- data[day == input$day]
: }
: if (input$hour != "All") {
data <- data[hour == input$hour]
: }
但是,最好的改法是下面这样:
filter <- ""
for (colname in c("year", "month", "day", "hour")) {
if (input[[colname]] != "All")
filter <- sprintf("%s %s %s == %s", filter,
ifelse(ncahr(filter)>0,"&",""), colname, input[[colname]])
}
data <- alldata_2010[eval(filter)]
: cPal <- colorNumeric(palette =
: c("green","orange","red","purple"),domain = 0:100)
: leaflet(taiwan) %>% addProviderTiles(providers$CartoDB.Positron) %>%
: addPolygons(color = "#444444", weight = 1.5, smoothFactor = 1.5,
: opacity = 1.5, fillOpacity = 0.1) %>%
: addCircleMarkers(lng=data$TWD97Lon,lat=data$TWD97Lat,
: radius=13,stroke=FALSE, fillOpacity = 0.9,
: fillColor = ~cPal(data$PM2.5),
: label =~as.character(data$PM2.5),
: popup = ~as.character(data$site))%>%
: addLegend("bottomright", pal = cPal, values =data$PM2.5,
: title = "PM2.5",
: labFormat = labelFormat(suffix = " "),opacity = 1)
: }
: )
: }
: shinyApp(ui = ui, server = server)
: [环境叙述]:
: R version 3.3.3
: windows 10
: 内存16gb
: [关键字]:shiny 资料大
作者: Esmelee (Esme)   2017-10-22 11:22:00
不行欸,还是没办法,错误写内存不足了cannot allocate vector of size 93.3 Mb修改memory.limit还是不行,我决定用sql试试
作者: psinqoo (零度空間)   2017-11-11 22:48:00
插话 windows 10 上做 shiny??

Links booklink

Contact Us: admin [ a t ] ucptt.com